-
WatchDjangoCon 2010: Switching addons.mozilla.org from CakePHP to Django
By
Featured 1 year agoSwitching addons.mozilla.org from CakePHP to Django
Presented by Jeff Balogh
In January of 2010 Mozilla started switching addons.mozilla.org (AMO) from CakePHP to Django. We see about 120 million web requests per month and 1.4 million visitors per day. This talk will be a case study of scaling and deploying a large website (and getting faster than PHP) with MySQL, memcached, virtualenv, Celery, Hudson, Redis, MongoDB, and more.
Abstract
To scale the backend of AMO, we rely heavily on caching in memcached (and soon Redis) to keep the load off our database, and multidb to spread the load we can't dump off on cache. Our caching ranges from object and query caching, to template fragment caching, up to full response caching. Frontend caching is external to Django and won't be covered. In addition, we work hard to measure and improve raw speed. Hitting indexes and keeping datasets small is critical, and the Django ecosystem has many great tools to help us profile.
We create development environments using pip and virtualenv, so we naturally started using that setup to deploy the site in production. We ran into issues creating a reliable environment with this method; switching to a separate vendor repository solved our problems with only a small amount of added effort. We update and deploy new code at least once a week, and have to deal with the additional complication of managing and interacting with parts of the PHP site that have not been ported over.
Apart from scaling and deployment, we've customized our application in ways that could be interesting to other Django users:
Jinja is our templating engine (but we keep the admin working) Babel helps us localize the site in over 30 languages Sphinx (not the documentation tool) serves our search results Celery and RabbitMQ help us do offline processing Nose and Hudson help us get over 90% code coverage in tests
-
WatchPyOhio 2010: Teach Me Python Bugfixing
By
Featured 1 year agoTeach Me Python Bugfixing
Presented by Catherine Devlin
Python wants YOU to help maintain and extend the language we love so much... but maybe you don't know how to go about making your contribution. Catherine doesn't either. David Murray, an experienced Python contributor, will guide her - and you - through the process, live and unscripted. Come take part in learner-controlled instruction. No C programming required!
-
WatchPyOhio 2010: PyPy and Unladen Swallow: Making your Python Fast
By
Featured 1 year agoPyPy and Unladen-Swallow: Making your Python Fast
Presented by Alex Gaynor
Python has a reputation for being a bit slow, but it doesn't have to be that way. This talk will cover why Python is slow, and what two of the most exciting virtual machines are doing about it.
PyGotham 2011: Machine Learning for Web Developers
Machine Learning for Web Developers
Presented by Al Barrentine
Machine learning deals with a class of algorithms which improve and evolve as they process more data. It has wide-ranging applications in recommendations, search, spam/fraud detection, facial recognition and other areas. The algorithms themselves will be covered but the real focus of this class is on how to use said algorithms in the web applications we work on every day. I&aposll try to keep the math and notation relatively light. Most of the algorithms you&aposll need to get started with machine learning are implemented for you in the various libraries. They comprise the "science" of machine learning and I hope you will decide to learn it, but mastering that material is a significant commitment of your time and mental energy (and has some additional prerequisites including a strong understanding of linear algebra). This class will focus on the "art" of machine learning, how to think about machine learning algorithms and integrate them into your web application.
Published 4 months ago
By
PyGotham 2011: Tmux + IPython = Awesome
Tmux + IPython = Awesome
Presented by Alexander Gaudio
This is a two part class aimed at expanding a developer's toolset with both tmux and IPython. In the first half, we will learn about tmux. I will first explain its client-server model and general architecture, compare it to GNU screen, and show some use cases. Then, we will start using it, create a customized profile, and solve a simple distributed computing problem with tmux. In the second half of this class, we will start using IPython. I will first explain what it is and review different use cases. Then, we will start using some basic commands, customize our profile and aliases, integrate python and shell scripting into the same code, and, time permitting, see some of the interactive scientific computing and interactive distributed computing possibilities. If you have one, bring a laptop with tmux and IPython already installed.
Published 4 months ago
By
PyGotham 2011: Building scalable websites and API's with Brubeck
Building scalable websites and APIs with Brubeck
Presented by James Dennis
In this talk I intend to teach people who know nothing about Brubeck how to build and deploy an entire site, providing all the commonly needed functionality we expect from other Python web frameworks.
Published 4 months ago
By
PyGotham 2011: Powerful Pythonic data analysis using pandas
Powerful Pythonic data analysis
Presented by Wes McKinney
In this talk I will give an overview on the pandas data analysis package for Python, its features, and plans for future development. I'll use various interesting data sets to illustrate the features and give motivation for how the tools can be applied in a diverse set of fields.Published 4 months ago
By
PyGotham 2011: Go Go Gadget Python
Go Go Gadget Python
Presented by Nicholas Waite
So you know that embedded devices are everywhere. Perhaps you've thought how nice it would be to make a Linux USB driver for some Windows-only device, or you've got something proprietary you would like to reverse-engineer and repurpose for your next big scheme. We know Python can do pretty much anything inside the computer--but how does a software person enter the world of circuits? And once you have some circuits, how can you bring the data back into your box? Bridging the worlds of hardware and software, I will show the power of PyUSB and Pyserial to pwn some sweet hardware and charm it over the USB port. From my own trials and tribulations building and hacking real devices, from a simple HID-class USB missile launcher to the custom protocol used in a complex biomedical data acquisition system, you will learn about USB packet sniffing, rapid-prototyping device drivers in python, and deciphering circuit boards and data sheets for fun & profit. I aim to leave you armed and ready to take on hardware of your own.
Published 4 months ago
By
DjangoCon 2009: Django is obsolete (but so is everything else)
Django is obsolete (but so is everything else)Presented by Avi Bryant
Keynote for DjangoCon 2009.
Published 2 years ago
By
PyCon 2010: Scrape the Web: Strategies for programming websites that don't expected it
Scrape the Web: Strategies for programming websites that don't expect itPresented by Asheesh Laroia
Do you find yourself faced with websites that have data you need to extract? Would your life be simpler if you could programmatically input data into web applications, even those tuned to resist interaction by bots?
Year by year, the web is becoming a stronger force. Learn how to get the best of it.
We'll discuss the basics of web scraping, and then dive into the details of different methods and where they are most applicable. You'll leave with an understanding of when to apply different tools, and learn about automating a full web browser, a "heavy hammer" that I picked up at a project for the Electronic Frontier Foundation.
Atendees should bring a laptop, if possible, to try the examples we discuss and optionally take notes. Code samples will be made available after class with no restrictions. Intended Audience
Intermediate (or better) Python programmers, probably without extensive web testing experience
Class Outline My motto: "The website is the API." Choosing a parser: BeautifulSoup, lxml, HTMLParse, and html5lib. Extracting information, even in the face of bad HTML: Regular expressions, BeautifulSoup, SAX, and XPath. Automatic template reverse-engineering tools. Submitting to forms. Playing with XML-RPC DO NOT BECOME AN EVIL COMMENT SPAMMER. Countermeasures, and circumventing them: IP address limits Hidden form fields User-agent detection JavaScript CAPTCHAs Plenty of full source code to working examples: Submitting to forms for text-to-speech. Downloading music from web stores. Automating Firefox with Selenium RC to navigate a pure-JavaScript service. Q&A; and workshopping Use your power for good, not evil.
Published 1 year ago
By
PyCon 2009: Scrape the Web: Strategies for programming websites that don't expect it (Part 1 of 3)
[VIDEO HAS ISSUES: Speaker walked away from the mic most of the time.] Do you find yourself faced with websites that have data you need to extract? Would your life be simpler if you could programmatically input data into web applications, even those tuned to resist interaction by bots? We'll discuss the basics of web scraping, and then dive into the details of different methods and where they are most applicable. You'll leave with an understanding of when to apply different tools, and learn about a "heavy hammer" for screen scraping that I picked up at a project for the Electronic Frontier Foundation. Atendees should bring a laptop, if possible, to try the examples we discuss and optionally take notes.
Published 2 years ago
By
PyCon 2011: web2py secrets tutorial (1/2)
web2py secrets
Presented by Massimo Di Pierro
We will provide an introduction to web2py with particular focus on its design objectives, its differences when compared to other web frameworks, and some of the most recently added features (components, plugins, openid/oauth integration).
Abstract
Web2py is a framework for rapid web application development of secure database-driven web applications. In this tutorial we will provide a brief introduction to web2py, the web based IDE, its internal design, and its internal file organization. We will explain the motivations behind some of its most controversial design goals. We will also discuss in some detail its Ajax capabilities by using web2py components and plugins for modular design of rich web interfaces.
This will be a very concentrated version (10%) of a course (CSC438) on web frameworks the author teaches at DePaul University. The course has been taught 3 times and the last time counted more than 30 students.
Published 10 months ago
By
PyCon 2010: Django Deployment Workshop
Django Deployment Workshop
Presented by Jacob Kaplan-Moss
So you've written a Django site... now what? Writing the application is just the beginning; now you've got to put it into production! In this hands-on workshop we'll walk through the creation of a full Django deployment environment running on a cluster of (virtual) machines.
This class will introduce students to a wide variety of technologies, including Amazon's EC2 and S3, Fabric, Varnish, nginx, mod_wsgi, memcached, PostgreSQL, pgpool, pg_standby, and more.
By the end of the class, each student will have created an environment featuring:
Multiple levels of caching and load balancing. Multiple web application servers. Shared media servers. Redundant replicated database servers. Separate caching servers.We'll create this whole stack on Amazon EC2 virtual machines, so students will be able to take home a complete copy of all the deployment configuration for future review at their leisure.
Note: the example application used in this class with be a Django site, and the class assumes at least a moderate comfort level with Django. However, the tools and techniques apply just as well to other Python web frameworks, so users of other tools will likely be able to follow along fairly easily.
Intended AudienceIntermediate to advanced Django users with sites nearing or in production. Python developers using other web frameworks will likely be able to follow along -- see the note in the summary, above.
Class Outline Introduction: the deployment question Application server: mod_wsgi Database server: PostgreSQL Automated deployment with Fabric Multiple web servers; load balancing with nginx Adding memcached Database connection middleware: pgpool Database redundancy: warm standby Upstream caching: Varnish Final load testing: how'd we do? RequirementsStudents will need a computer and Amazon EC2 and S3 accounts, and will be asked to familiarize themselves with the process of starting and stopping EC2 virtual machines before the class.
http://us.pycon.org/2010/tutorials/kaplanmoss_django/Published 1 year ago
By