As wordpress is blocked by the Wall in China, now my new blog posts currently are written in Zhihu Column: 技术宅
由于众所周知的原因,中国内地无法访问wordpress站点,目前本博客新增内容转到知乎专栏 技术宅
各位iOS开发同学们,你们打出来的iOS的ipa包真的已经不能再小了么?如果你也像题图一样,.ipa
文件比AppStore上显示的大小相差太多,那你应该对我说的有兴趣。
以下是从一个QA的角度作出的从发现问题到解决问题的整个详细历程,对ObjC的编译连接方式可能理解是错误的,还请各位ObjC大牛指正。文章较长,我尽量讲得更清楚些。为了防止大家对这种长文直接放弃,我先上一个优化前后对比结果图:
以阿里旅行5.2.0发布到Appstore的版本为例,对比结果如下:
对比 | 优化前 | 优化后 | 减少了 |
---|---|---|---|
二进制文件大小 | 46.6M | 31.7M | 14.9M |
.ipa文件大小(本身就是zip格式) | 25.2M | 22.3M | 2.9M |
手机“用量”中显示的大小 | 57.2M | 43.0M | 14.2M |
注:前两项大小都是Mac OS X 10.10中,通过命令行ls命令查询得到
如果你还是没兴趣看长篇大论,直接看最后“我找到的优化方法”那段就行。
为了方便大家理解,先对Xcode生成的各种包进行说明:
iOS编译以后生成的原始文件,实际是一个文件夹,里面包含各种资源文件(图片,第三方bundle,plist等文件),程序的可执行文件(二进制格式)以及对所有文件的签名记录(_CodeSignature
)
不能上传AppStore
生成.app
时的附属产物。本质是一个文件夹,其中只有一个最大的文件,作用是对iOS程序闪退后产生的log文件进行符号化(desymbolicate);通俗的说,就是把无意义的内存地址变成可读的程序中的类和方法以及代码行数
不能上传AppStore
实际上就是把.app
放到Payload
文件夹后,对Payload
就行了zip操作,最后改了下扩展名。
可通过Application Loader
上传AppStore
实际上也是一个文件夹,包含.ipa
和.dSYM
文件
可通过Xcode
上传AppStore
由于上传AppStore速度很慢,我们一般选择上传.ipa
文件到AppStore。那为什么从苹果商店看到的包大小和我们的.ipa
大小不一致呢。下面是苹果官方的说法(注意黑体字,后面会用到):
When your application is approved by Apple to sell on the App Store, it is encrypted for DRM purposes and re-compressed. When the encryption is added, the size of the compressed file will increase. The exact size of the increase will vary from app to app, however, the size increase can be large when the binary contains a lot of contiguous zeros. We are unable to guarantee the size of your file after the encryption has been added.
也就是说,苹果获取上传的ipa文件后,进行解压缩成.app
,然后对其中二进制文件进行Apple FairPlay DRM加密,最后重新压缩成.ipa
,此时生成的.ipa
作为AppStore上的显示程序的大小。
回到我们阿里旅行的包:我们v5.1.1
生成的.ipa
文件是23.9M左右,而苹果商店上显示的是36.9M
为什么会有这么大差距呢,我们来看看我们的二进制文件AliTrip
的前后大小对比:
adding: Payload/Alitrip.app/AliTrip …. (in=42806208) (out=14899495) (deflated 65%)
上面是打包机使用PackageApplication
对.app
进行压缩成zip(即ipa)时的日志。可以看到我们的二进制文件高达42.8M,经过zip压缩后变为14.8M左右,缩小了65%的体积。
为什么我们的文件压缩比率可以这么高呢。我们使用Sublime Text打开此文件就会发现,果不其然!连续的0数都数不过来。
以上猜想可以通过iTunes下载ipa文件来看:
通过iTunes下载阿里旅行,找到下载到的ipa文件,解压缩后,对其中的Payload
文件夹中进行zip压缩,发现二进制的压缩比与之前相比,已经下降了30%:
adding: Payload/Alitrip.app/AliTrip (deflated 35%)
从42806064压缩到27654347
所以DRM操作是导致.ipa
文件变大的原因
我们现在要做的就是如果减少和消除我们二进制文件中多出的哪些连续的0:
对与iOS的编译和连接等操作我完全是门外汉,经过几天的搜索整理,我找到的突破口就是 Strip 。strip
从字面意思上其实就是“脱光”的意思(嗯,点题了…),也就是把生成的 对象文件 (.o
文件)中不必要的 符号(symbols)去掉的意思。所以从Xcode的Strip
相关配置下手进行优化。以上是我对strip的理解,可能不完全对,望大家指正。
在Xcode项目的Build Settings
中,搜索Strip
:我们可以看到,程序默认做了一些配置,相关strip选项解释如下(BuildSettings所有选项的官方点这里)。
选项 | 意义 |
---|---|
Deployment Postprocessing | strip所有选项的总开关,如果选NO,以下选项均无效 |
Strip Debug Symbols During Copy | 文件拷贝编译阶段时是否进行strip,你的工程中有CopyFilesBuildPhase才有意义 |
Strip Linked Product | 这个选项才对最后生成的二进制文件进行strip |
Strip Style | all ,non-global ,debugging strip程度依次降低:all 一般用于最后生成.app的工程;non-global 用于bundle和framework,debugging 一般都可以。虽然all 是strip最多的选项,但是选择错误会导致strip失败 |
Dead Code Stripping | 用于删除对象文件中不需要加载的符号,减小二进制文件大小 |
这是阿里旅行其中两个工程的strip设置截图(其中 粗体 是优化过的选项):
可以看到阿里旅行的strip设置基本正确,但是最关键的strip总开关Deployment Postprocessing
没有打开。然后就是Strip Style
对线上包的strip程度不够(framework工程只选择了debugging
级别,改为non-global
)
这个strip总开关在Xcode的Release配置中本身也是默认关闭的,需要我们手动打开。另外其实strip除了降低app大小外,一定程度上提高了从app获得更多信息的难度,安全性更佳。
以上strip优化只针对内测包和商店包。
由于iOS工程的线上配置中使用了Xcode的默认设置,导致没有开启strip开关,最后生成的二进制文件偏大。
由于我对ObjectiveC编译、连接的实现完全不理解,我这种方式不一定是最优化的选择;也可能除了strip外,BuildSetting还有其他的优化方式。在此抛砖引玉,欢迎大家讨论。
最后,这种处理后会不会对我们crash日志解析产生影响,还需要开发同学来确认下。优化过的app可以正常运行,我已经试验过了。
The Jenkins Clang Scan-Build Plugin have not been updated for years. I fixed existing bugs to make it usable in Xcode 5,6 and latest clang-static-analyzer. I have published on Github, you could check out for more details.
scan-build
and xcodebuild
additional arguments
--use-analyzer Xcode
in scan-build additional arguments: this would make scan-build use clang executable from Xcode instead of its own-derivedDataPath $WORKSPACE/build
in xcodebuild additional arguments: this will save build products and other derived data to ‘build’ directory under current worksapcegit clone https://github.com/truebit/clang-scanbuild-plugin.git
cd clang-scanbuild-plugin
mvn clean package -Dmaven.test.skip=true
clang-scanbuild-plugin.hpi
in target
directory"Manage Plugins">"Advanced">"Upload Plugin"
Any concerns, you could fire an issue 🙂
If you are an Objective C developer using Xcode, and push your code to Git/SVN like other guys did in the team, I think you most probably have encountered the merge conflicts of project.pbxproj
file.
It’s such a pain to merge this file by searching the file with <<<
,>>>
and ===
, and then deleting and keeping lines by your judge. It would leave some unused lines or even make Xcode build fail due to wrong decision.
Since UUID generated by Xcode in project.pbxproj
file is not unique for all machines, different Xcode got different UUID for the same file or filegroup. That’s why it created conflicts.
I just found that Xcode does not care the UUID in the file, it just needs to be unique in the file. So I made xUnique to fix the merge conflicts issue.
sort-Xcode-project-file
, supports following new features:
PBXFileReference
and PBXBuildFile
sectionsgit add path/to/xUnique.py
, so all members could use the same scriptln -s path/to/xUnique.py .git/hooks/pre-push
chmod 555 .git/hooks/pre-push
pre-push
instead of pre-commit
is a safe consideration: you decide to commit the newly generated project file or notproject.pbxproj
file in either way:
python path/to/xUnique.py path/to/MyProject.xcodeproj
and then committing changes.最近用bottle写后台api接口,返回的都是json格式。后来写前端的同学说不能解析json,然后搜了下终于知道有JSONP这个东东,然后知道jsonp的格式原来就是一个callback函数名称xxx(请求中带有callback=xxx)包着一个json数据的方法的形式。其实JSONP只是用来跨域通信用的,虽然现在的需求不需要用jsonp,不过某同学没有研究出来js里面怎么直接获得同一域JSON数据,所以我还是折腾了一下。挺好,又学到些东西:P
直接上代码:如果使用的是default_app,则直接把下面代码贴进对应的py文件中就ok了
from bottle import response, request, install def jsonp(callback): def wrapper(*args, **kwargs): resp = callback(*args, **kwargs) if isinstance(resp, (dict,list)): #response.charset='utf-8' # set property error , do not know why response.content_type = 'application/json;utf-8' # after added this line, we do not need to mess with character encoding in json.dumps; the commented out code is what i did before callback_arg = request.query.get('callback') if callback_arg: resp= '{}({})'.format(callback_arg, json.dumps(resp))#,ensure_ascii=False))#.encode('utf-8') return resp return wrapper install(jsonp) # install the plugin in the bottle app
另外就是发现,处理jsonp为这样的字符串以后,返回的结果数据的中文直接显示为unicode的字符串形式(\uxxxx形式)。解决方法:
为了造福世界人民,我觉得还是用英语写吧 XD
Today I got a request to write a backend server to provide interface access from frontend. I determined to use Python and RESTful api firstly . I hate writing Java code…
After some investigation, I picked up Bottle. It is very neat and small. And I also read that with nginx and uWSGI, the performance is good too.
My environment is Ubuntu 12.04.1 LTS; but after followed several posts, my environment still did not work. I want to share my findings to save others’ time:)
Let’s cut to the chase:
My application would be in below directory constructure, all following commands and configurations are based on below:
/var/www/myapp would be the root directory of my bottle app
/var/www/myapp/env would be the virtualenv for the app
/var/www/myapp/index.py would be main .py file of bottle
sudo apt-get update sudo apt-get install -y nginx uwsgi sudo apt-get install uwsgi-plugin-python
sudo apt-get install python-pip
sudo pip install virtualenv
sudo mkdir -p /var/www/myapp sudo virtualenv /var/www/myapp/env source /var/www/myapp/env/bin/activate pip install bottle deactivate
sudo chown -R www-data:www-data /var/www/myapp
1. nginx config:
sudo gedit /etc/nginx/sites-enabled/default
copy below config in opend file:
server { listen 80; charset utf-8; root /var/www/myapp; server_name localhost; location / { include uwsgi_params; uwsgi_pass unix:/tmp/uwsgi.myapp.socket; uwsgi_param UWSGI_PYHOME /var/www/myapp/env; uwsgi_param UWSGI_CHIDIR /var/www/myapp; uwsgi_param UWSGI_SCRIPT index; # this should be the .py file name without suffix that your bottle will use to launch } }
2.uWSGI config:
sudo gedit /etc/uwsgi/apps-enabled/uwsgi.ini
copy below config in opend file:
[uwsgi] plugins=python socket=/tmp/uwsgi.myapp.socket pythonpath=/var/www/myapp
Put index.py under /var/www/myapp
#!/usr/bin/env python from bottle import route, run, default_app @route('/') def index(): return "Aloha, world~" if __name__ == "__main__": run(host="localhost", port=8081) else: application = default_app()
sudo service nginx restart sudo service uwsgi restart
Now “it’s the moment to witness the miracle!” 😀 Access http://localhost/ in your browser, you should see “Aloha, world~”. If not, comment below:)
Oct/7/2012 update:
I just wrote this for fun and did not want to be in any trouble so I did not upload the script. But it seems that many guys really typed it character by character, I think I should upload it 🙂
Here is the download link. Have fun. By the way, my battle tag is SeanWang#1150, usually play on US Server, also Asia Server sometimes.
After I noticed that guys posted that certain servers of Diablo 3 have good drops (orginal post and battlenet quote) than others, I observed this ip address stuff when repeatedly running Warrior’s Rest using a monk with all MF items
My conclusion is that it is some sort of correct! especially the part about ” last number of the ip address that is over 80 is not good”.
certain servers have good drops and some have bad drops, each time you log in you are connected to a random server this is how you can find servers with better drops than others.
1) go to your cmd promt while logged into a game (not at the log in screen or at char screen but in a game)
2) type in “netstat -n” into your cmd promt to find what server you are connected to the last 4 digits of the ip are what matter. anything with 3 digits sucks and typically games with 2 digit ending in even numbers are good loot games. The best ones are “74:1119” and “76:1119” but 72-78 even work also, games like 46 also work.
So based on this theory, I wrote a Sikuli script to get the ip address less than 80. In my script, I did not judge for even number, because I think that it may be too long to get a even number AND less than 80. Too many times creating and quiting games may trigger the Blizzard anti-bot mechanism…
Here comes the script:)
NOTICE: Above script should only work in windowed game mode.
Have fun:)
找到个免费VPN不容易,虽然过一会就断线,但是对于俺连接美服更新Diablo3是绰绰有余了。不过这个VPN需要每3天登录一次,这个比较麻烦。
思路:
开工:
#!/usr/bin/env python #-*- coding: UTF-8 -*- # filename: AutoLogin.py from __future__ import unicode_literals import urllib2 import cookielib import urllib import Image from cStringIO import StringIO import re from pytesser import * LOGIN_URL = 'http://*.*.*.*/lr.sm' #网站就隐了,被发现了估计验证码加强了就不好整了-_-|| IMAGE_URL = 'http://*.*.*.*/image' USER = 'yourusername' PWD = 'yourpassword' ### OCR using pytesser ### img_file=urllib2.urlopen(IMAGE_URL) img= StringIO(img_file.read()) checkImg= Image.open(img) ocr_str= image_to_string(checkImg).lower() CODE=''.join(ocr_str.split()) postdata=urllib.urlencode({ 'user.nick':USER, 'password':PWD, 'validationCode':CODE, }) headers={ 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1', 'Referer':LOGIN_URL } cookie_support = urllib2.HTTPCookieProcessor(cookielib.CookieJar()) opener = urllib2.build_opener(cookie_support, urllib2.HTTPHandler) urllib2.install_opener(opener) req = urllib2.Request( url = LOGIN_URL, data = postdata, headers = headers ) result = urllib2.urlopen(req).read() decoded_result=result.decode('utf-8') if re.search('{} **欢迎您'.format(USER), decoded_result): #隐去网站名称... print 'Logged in successfully!' else: with open('result.html','w') as f: f.write(result) print 'Logged in failed, check result.html file for details'
应该是只登录就好了,所以没对cookie做处理。以后有时间研究下cookielib~
前几天写的“你画我猜”(draw something)单词猜测工具,最后提到了有道已经做了相关工具。既然有大树了,俺们就可以直接乘凉了~
使用有道网页版的改进版本来了:直接发送查询请求然后对返回结果格式化就搞定了,非常easy:)(需要用到lxml模块)
#!/usr/bin/env python # -*- coding: utf-8 -*- """ @author: Sina Weibo @SeganW """ from __future__ import unicode_literals import urllib import lxml.html as lh import sys def query(char_str,num=None): if not None: num=len(char_str) con=None resp=None try: con=urllib.urlopen('http://dict.youdao.com/drawsth?letters={}&length={}'.format(char_str,num)) resp=con.read() except Exception as e: if con: con.close() raise SystemExit('[error] guess request failed: {}'.format(e)) doc=lh.document_fromstring(resp) words=[i.text_content().lower() for i in doc.xpath("//span[@class='word']")] translates=[i.text_content() for i in doc.xpath("//div[@class='trans']")] return zip(words,translates) if __name__ == '__main__': args=sys.argv if len(args)==2: charstr= args[1] num=None elif len(args)==3: charstr= args[1] num=int(args[2]) else: raise SystemExit('usage: drawsomthing.py charstr [num]') results=query(charstr,num) if results: for sub_result in results: print '{}:\t{}'.format(*sub_result) else: print "No results found:("
使用方法不变:
运行的命令行就是
drawsomething_updated.py char_str [num]char_str为程序显示的乱序字符串,把让你拆的所有字符串起来做一个字符串
num是程序让你猜的单词包含的字符个数;可不填,这样就变成python版词典了:)
今天才用上正统的Appstore软件下载方法:从iTunes端下载,然后PC端与iOS设备同步。然后俺就发现,下载速度那个慢啊。。。
从网上得到如下消息:
苹果为 App Store 准备了 a1.phobos.apple.com 到 a2000.phobos.apple.com 这么一群服务器,美国、欧洲、日韩、港澳……就是没有中国。。。
不管真假,让咱们用Python挨个ping出哪个地址最快,然后加到hosts里面吧:)
以下代码为Windows中文版,如果其它版,请把speed_ptn行的字符串改成对应的:
Nov/4/2014更新:
multiprocessing.dummy处理I/O负荷的并发,multiprocessing处理CPU负荷的并发。多少个url都无所谓啦~
注意:只写了Mac OS X版本的
#!/usr/bin/env python # -*- coding: utf-8 -*- """ @author: Sina weibo @SeganW Only works in Mac OS X, I tested in Yosemite """ from __future__ import unicode_literals from subprocess import check_output, CalledProcessError from re import compile as re_compile from sys import getfilesystemencoding from os import devnull from multiprocessing.dummy import Pool URLS = ['a{}.phobos.apple.com'.format(i) for i in range(1,2001)] IP_PTN = re_compile('\d+\.\d+\.\d+\.\d+') AVG_PTN = re_compile('min/avg/max/stddev = \d+\.\d+/(\d+\.\d+)/') def ping_test(url): try: with open(devnull, 'w') as null: output = check_output(['ping','-c','3', url], stderr=null).decode(getfilesystemencoding()) ip = IP_PTN.search(output).group() avg = float(AVG_PTN.search(output).group(1)) except CalledProcessError: ip = "" avg = "" print 'Server {}\t||\t Average Speed: {}ms'.format(url,avg) return {ip: avg} if __name__ == '__main__': pool = Pool(25) #25 is process count, try increase or decrease it to find the best performance results = pool.map(ping_test, URLS) pool.close() pool.join() best = min(results,key=lambda x: x.values()[0]) best_svr = results.index(best) print '***** Best server is a{}.phobos.apple.com, ip={}, avg speed={}ms *****'.format(best_svr, best.keys()[0],best.values()[0])
Aug/20/2012更新:写了使用多线程并加入了对timeout情况的处理,发现虽然速度快了,但是结果却是每次都是最大那个数的服务器ping值最快。不知道为啥。另外并发数不能太大,URLS里面直接用2000会提示打开文件太多。改成500没发现问题。
新版代码如下:
#!/usr/bin/env python # -*- coding: utf-8 -*- """ @author: Sina weibo @SeganW """ from __future__ import unicode_literals import subprocess import re import sys import threading URLS=map(lambda x: 'a{}.phobos.apple.com'.format(x), range(1,501)) NUM=0 BEST=[2**16,'',''] # a big number normally larger than average IP_PTN=re.compile('\[(.*)\]') SPEED_PTN=re.compile('平均 = (\d+)ms') class Ping(threading.Thread): def __init__(self, url, lock): threading.Thread.__init__(self) self.url=url self.lock=lock def run(self): try: output=subprocess.check_output(['ping',self.url]).decode(sys.getfilesystemencoding()) except subprocess.CalledProcessError as cpe: print cpe.output output=cpe.output.decode(sys.getfilesystemencoding()) avg='timed out' else: avg=&amp;quot;{}ms&amp;quot;.format((SPEED_PTN.search(output).group(1))) ip= IP_PTN.search(output).group(1) self.lock.acquire() print 'Server {}\t||\t Average Speed: {}'.format(self.url,avg) if avg =='timed out': pass elif BEST[0] &amp;gt; int(avg[:-2]): BEST[0] = int(avg[:-2]) BEST[1] = ip BEST[2] = url global NUM NUM+=1 if NUM==len(URLS): print 'Fastest server: {2} ( {1} ) fastest average speed: {0}ms'.format(*BEST) self.lock.release() if __name__ == '__main__': lock=threading.Lock() try: for url in URLS: ping_thread=Ping(url,lock) ping_thread.start() except KeyboardInterrupt: pass
原版:
#!/usr/bin/env python # -*- coding: utf-8 -*- """ @author: Sina weibo @SeganW """ from __future__ import unicode_literals import subprocess import re import sys urls=map(lambda x: 'a{}.phobos.apple.com'.format(x), range(1,2001)) best=[2**16,'',''] # a big number normally larger than average ip_ptn=re.compile(r'\[(.*)\]') speed_ptn=re.compile(r'平均 = (\d+)ms') try: for url in urls: output=subprocess.check_output('ping {}'.format(url),shell=True).decode(sys.getfilesystemencoding()) output=[i for i in output.split('\r\n') if i] ip= ip_ptn.search(output[0]).group(1) avg=int(speed_ptn.search(output[-1]).group(1)) print 'Server {}\t||\t Average Speed: {}ms'.format(url,avg) if best[0] &amp;gt; avg: best[0] = avg best[1] = ip best[2] = url except KeyboardInterrupt: print '\n' print 'Fastest ip address:\n{1} {2}\nfastest speed: {0}ms'.format(*best)
找到最快的以后就可以改hosts文件了,这个不懂的自己google吧。