Python 爬取YouTube某个频道下的所有视频信息 被生活安排了这么一个需求。 需要爬取YouTube给的频道下的给定日期范围内的视频的信息，如标题，点赞数，点踩数，播放量等信息 首先需要一个谷歌账号，翻墙工具来科学上网，打开YouTube，搜索指定的频道，进入频道界面 然后查看网页源代码，搜索channel，得到频道的频道ID 然后还需要申请谷歌数据API的个人独有的API key，参照博客申请api key并指定YouTube api 下面是谷歌的api地址 self.app_key = '你自己的 api key'self.channel_api = 'https://developers.google.com/apis-explorer/#p/youtube/v3/youtube.channels.list?part=snippet,contentDetails&publishedAfter=2016-11-01T00:00:00Z&publishedBefore=2017-12-31T00:00:00Z&id='+ channel_id + '&key=' + self.app_keyself.info_api = 'https://www.googleapis.com/youtube/v3/videos' 本来的思路是找出所有的视频地址，然后根据视频发布日期过滤结果，而恰巧谷歌限制了API的返回结果为500个（实际为500个左右），导致视频缺失，导致我思考了很久解决办法，最终还是Google到了结果（Google google的问题 =  =） 相关摘录： “如果没有搜索结果的质量严重降低(重复等),我们无法通过API为任意YouTube查询提供超过500个搜索结果. v1 / v2 GData API在11月更新,以限制返回到500的搜索结果数.如果指定500或更高的起始索引,则不会获得任何结果. 因此，为了获取全部指定时间段发布的视频，需要在参数里加上发布日期界限（分时间段搜索，每次的搜索结果上限仍然是500，请特别注意！！！！） publishedAfter=2016-11-01T00:00:00Z&publishedBefore=2017-12-31T00:00:00Z 下面贴完整代码： # -*- coding: UTF-8 -*- import urllib2 import time import urllib import json import datetime import requests import sys import xlsxwriter reload(sys) sys.setdefaultencoding("utf-8") channel = "Samsung"#频道名 channel_id = 'UCWwgaK7x0_FR1goeSRazfsQ'#频道ID class YoukuCrawler: def __init__(self): self.video_ids = [] self.maxResults = 50#每次返回的结果数 self.app_key = '你自己的 api key' self.channel_api = 'https://developers.google.com/apis-explorer/#p/youtube/v3/youtube.channels.list?part=snippet,contentDetails&publishedAfter=2016-11-01T00:00:00Z&publishedBefore=2017-12-31T00:00:00Z&id='+ channel_id + '&key=' + self.app_key # self.info_api = 'https://www.googleapis.com/youtube/v3/videos?maxResults=50&part=snippet,statistics' + '&key=' + self.app_key self.info_api = 'https://www.googleapis.com/youtube/v3/videos' now = time.mktime(datetime.date.today().timetuple()) def get_all_video_in_channel(self, channel_id): base_video_url = 'https://www.youtube.com/watch?v=' base_search_url = 'https://www.googleapis.com/youtube/v3/search?' first_url = base_search_url + 'key={}&channelId={}&part=snippet,id&publishedAfter=2016-11-01T00:00:00Z&publishedBefore=2017-12-31T00:00:00Z&order=date&maxResults=25'.format(self.app_key, channel_id) url = first_url while True: print url request = urllib2.Request(url=url) response = urllib2.urlopen(request) page = response.read() result = json.loads(page, encoding="utf-8") for i in result['items']: try: self.video_ids.append(i['id']['videoId'])#获取作品ID except: pass try: next_page_token = result['nextPageToken']#获取下一页作品 url = first_url + '&pageToken={}'.format(next_page_token) except: print "no nextPageToken" break def main(self): self.get_all_video_in_channel(channel_id) return self.get_videos_info() def get_videos_info(self):#获取作品信息 url = self.info_api query = '' count = 0 f = open(channel_id + '.txt', 'w') print len(self.video_ids) for i in self.video_ids: try: count += 1 query = i results = requests.get(url, params={'id': query, 'maxResults': self.maxResults, 'part': 'snippet,statistics', 'key': self.app_key}) page = results.content videos = json.loads(page, encoding="utf-8")['items'] for video in videos: try: like_count = int(video['statistics']['likeCount']) except KeyError: like_count = 0 try: dislike_count = int(video['statistics']['dislikeCount']) except KeyError: dislike_count = 0 temp = time.mktime(time.strptime(video['snippet']['publishedAt'], "%Y-%m-%dT%H:%M:%S.000Z")) dateArray = datetime.datetime.utcfromtimestamp(int(temp)) otherStyleTime = dateArray.strftime("%Y-%m-%d") print otherStyleTime,count if (otherStyleTime>='2016-11-01' and otherStyleTime

如何使用Peertube托管自己的Youtube Peertube is a federated and open source video hosting platform that you can run from your own computer. Iit allows you to completely control all the content that you host and share from your website. This tutorial shows you how to install and host Peertube on Ubuntu.CONTENT Why Host and Use Peertube Installing Peertube Configuring Nginx and SSL Configuring and Running Peertube Using Peertube Frequently Asked Questions Tip: if you just want to watch YouTube offline, there is no need to install Peertube. Check out all the ways to watch YouTube offline.Why Host and Use PeertubeOne of the most attractive features of Peertube is its ability to load videos from other instances, making it possible to view content from outside your website but still retain the control over your data.Another advantage of Peertube over Youtube is that it is entirely open source. (Learn all about open source licenses here.) This makes it easy for anyone to scrutinize the program’s codebase, which can be helpful for users that are concerned about their data security.Good to know: learn more about protecting your data online by installing privacy and security extensions in Chrome.Installing PeertubeBefore you can install Peertube, you need to make sure that you have a server ready. This could be your personal PC or a rented server from a web host. This tutorial is done on an Ubuntu VPS from Digitalocean. Set up a new user account for Peertube. This will allow you to easily control what the program can do inside your system: sudo useradd -b /bin/bash -m -d /var/www/peertube -G sudo peertube sudo passwd peertubeCreating a new user account also allows you to set the $HOME variable under “/var/www/.” This is important, as the Web backend for Peertube will not be able to traverse the default “/home” heirarchy. Switch to your new user account with the command: su peertube Install the dependencies for Peertube: sudo apt install cron wget curl unzip python3-dev python-is-python3 certbot nginx python3-certbot-nginx ffmpeg postgresql postgresql-contrib openssl g++ make redis-server git Install NodeJS in your machine: curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash - sudo apt update sudo apt install nodejs Install Yarn. This is a powerful yet lightweight package manage for NodeJS: curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null echo "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list sudo apt update sudo apt install yarn Once you have installed all of the dependencies for Peertube, configure your system to install the program. First, enable the program’s database backend: sudo systemctl enable postgresql sudo systemctl start postgresql sudo systemctl enable redis-server sudo systemctl start redis-server Add your Peertube user to PostgreSQL by running the following commands: cd $HOME sudo usermod -aG peertube postgres sudo -u postgres createuser -P peertube Create the database for the program by running the following commands: sudo -u postgres createdb -O peertube -E UTF8 -T template0 peertube_run sudo -u postgres psql -c "CREATE EXTENSION pg_trgm;" peertube_run sudo -u postgres psql -c "CREATE EXTENSION unaccent;" peertube_run Create the directory structure of the program in your home directory: mkdir config storage versions chmod 750 ./config Download the Peertube binary files: cd ./versions wget https://github.com/Chocobozzz/PeerTube/releases/download/v5.0.1/peertube-v5.0.1.zip unzip peertube-v5.0.1.zip cd ./.. Create a symbolic link between your install and your home directory: ln -s /var/www/peertube/versions/peertube-v5.0.1 /var/www/peertube/peertube-latest Install Peertube using the following Yarn command: cd ./peertube-latest yarn install --production --pure-lockfileConfiguring Nginx and SSLBy default, Peertube opens its Internet service on port 9000. While you can access the program from that, it is good practice to create a reverse proxy between the program and a well-known port.The program’s developers have made a template file that you can use to create your own reverse proxy by running the following command:sudo cp /var/www/peertube/peertube-latest/support/nginx/peertube /etc/nginx/sites-available/peertube sudo rm /etc/nginx/sites-enabled/defaultConfigure your new template file by opening it using a text editor:sudo nano /etc/nginx/sites-available/peertubeInside, change every instance of these two variables: ${WEBSERVER_HOST} and ${PEERTUBE_HOST}. For the ${WEBSERVER_HOST}, replace it with your machine’s FQDN. Meanwhile, replace ${PEERTUBE_HOST} with “127.0.0.1:9000.” Press Ctrl + O, then Ctrl + X to save your file to disk and exit the text editor.Enable the Peertube Nginx config file and restart Nginx.sudo ln -s /etc/nginx/sites-available/peertube /etc/nginx/sites-enabled/ sudo systemctl reload nginxCreating Your SSL CertificateWe are obtaining a new SSL certificate using the free Certbot utility from Let’s Encrypt. (You can also create a wildcard SSL certificate if you intend to use it on multiple (sub)domains.)sudo certbotCertbot will scan your Nginx configuration and bring up the list of domains hosted on your server. Enter the number beside the domain for which you want to obtain a new SSL certificate.Once the SSL certificate is issued, certbot will auto-update your Nginx config file with the correct entry. You just need to reload your Nginx configuration to make sure all is running well.sudo systemctl reload nginxTip: enabling SSL will encrypt all TCP connections to your instance; however, it is better practice to secure your Linux server from the get go.Configuring and Running Peertube With both your Nginx server and SSL certificate done, you can now configure your Peertube instance. You can use a template that the developers have made to streamline this process. Run the following commands: cd $HOME cp /var/www/peertube/peertube-latest/config/default.yaml /var/www/peertube/config/default.yaml cp /var/www/peertube/peertube-latest/config/production.yaml.example /var/www/peertube/config/production.yaml Open the “production.yaml” file in a text editor: nano /var/www/peertube/config/production.yaml Change the hostname: variable to your machine’s FQDN: Generate a random secret for your instance with the following command: openssl rand -hex 32Go back to your “production.yaml” file and paste your random secret beside the peertube: variable. Look for the database: block. Change the suffix: block to “_run.” Change the password: variable to your database account’s password. Go to the smtp: block and find the hostname: variable. Change that to the hostname of your mail server. Also, change both the username: and password: variables to the credentials of your email account. Replace the from_address: variable with the email address of your email account. Once you are done making the changes, press Ctrl + o to save the file and Ctrl + x to exit the file.Creating a Peertube Service FileTo make Peertube run automatically at startup, we are creating a systemd service file for Peertube. Run the following command to copy the template systemd file to the system: sudo cp /var/www/peertube/peertube-latest/support/systemd/peertube.service /etc/systemd/system/ Reload systemd to apply your new service file. sudo systemctl daemon-reload sudo systemctl enable peertube sudo systemctl start peertubeUsing PeertubeIf everything is configured properly, you should be able to access Peertube from your own domain name.By default, every new Peertube instance creates a root account that you can use as the site’s administrator. To use this, run the following command:sudo journalctl -u peertube | grep "User password:"Go back to your Peertube website and press the “Login” button on the page’s upper-left corner. Write “root” as your username and paste its password.Peertube will greet you with a brief message that contains links to the program’s documentation.Once you have reviewed the content of the message, press X on the window’s upper- right corner to start using your Peertube website.Frequently Asked QuestionsIs it possible to use Peertube without a domain name?No. Peertube requires you to have a valid SSL certificate in your instance. While it is possible to create your own SSL certificate without a domain name, doing this will make your site insecure for other users.Can I copy the default.yaml file while configuring Peertube?Peertube depends on the “default.yaml” file for some of its core settings. Without the “default.yaml” file, your instance will most likely render it inaccessible.Why am I getting a blank page when I open my Peertube website?This issue is most likely due to a permissions issue with your root peertube directory. By default, Nginx requires every Web folder, as well as its root, to be world readable.You can fix this issue by running the following command: sudo chmod 755 /var/www/peertube.Image credit: Unsplash. All alterations and screenshots by Ramces Red.GitHub – Chocobozzz/PeerTube: ActivityPub-federated video streaming platform using P2P directly in your web browser