Dragonfly, an open-source python framework for scalable and robust Bayesian optimization, is developed by researchers from Carnegie Mellon University, Pittsburgh : Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing. The paper was submitted to Journal of Machine Learning Research in April 2020 titled “ Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly.
Machine Learning tasks are cast as black-box problems, where we have given some input and black-box does some processing and outputs the value for the given input. To enhance the performance of such a black-box system, we tend to optimize it to its optimum. An example of it is Hyperparameter Tuning where we search for the best parameters to find the best model for a given task. According to domain knowledge, this searching is often done by grid search or random search or manual settings(according to the domain knowledge). Formally, we define a black box as a function f : X → R over some domain X and try to find the optimum value of f via evaluating it repeatedly. Each evaluation, in this case, is expensive and requires huge computation. That’s when Bayesian Optimization(BO) comes into the picture. BO maximises the function f with a few evaluations by calculating the next point’s evaluation via previous observation. In probability terms, in order to determine the next evaluation, given the current evaluations as a point to start for the next evaluation, BO uses posterior. BO also showed some promising results in some hyperparameters tasks. Optimization tasks in today’s applications are posed to new threats that are difficult to tackle by traditional approaches. Dragonfly is primarily focused on scalable Bayesian Optimization and its robustness.
• Scalability: This includes handling higher dimensional domains, multi-fidelity evaluations, optimising over neural network architectures, handling parallel evaluations. Apart from this, Dragonfly also contains some evolutionary algorithms that are capable of optimizing complex domains.
• Robustness: Includes a set of values for a set of acquisition choices and model parameters instead of one for the entire optimization process. Hence, lowering the computational cost and adding robustness to the optimization system.
Requirements & Installation
Dragonfly is supported by Python 2(>=2.7) and Python 3(>=3.6) as well. It is fully compatible with Mac, Linux & Windows: You can install dragonfly via PyPI.
pip install dragonfly-opt -v
To check the installation of Dragonfly by following code :
#installation testing from dragonfly import minimise_function # The first argument below is the function, the second is the domain, and the third is the budget. min_val, min_pt, history = minimise_function(lambda x: x ** 4 - x**2 + 0.1 * x, [[-10, 10]], 10); print(min_val, min_pt)
Quick Start with Dragonfly
Dragonfly is supported by command line, python and ask-tell mode. This guide will contain all of the three approaches to dragonfly.
1. Dragonfly with Command line
Since, we can run command line on Colab Notebook, this demo contains the code for colab command line. Dragonfly can be used explicitly by calling dragonfly-script.py or by importing python code through maximize_function in the main library. Examples are given here in this directory. If you have installed dragonfly via pip, you have to clone the git repository to access the examples.
To use dragonfly through the command line, specify the optimization problem and the optimization parameters(can be specified in JSON (recommended) or protocol buffer format). You can check examples/synthetic/branin/config.json and examples/synthetic/branin/config.pb for examples. Then, specify the optimisation parameters in an options file, in the format shown in examples/options_example.txt.
Basic Usage : The example shown below is the Branin benchmark for global optimization.
%cd examples !python /content/dragonfly/bin/dragonfly-script.py --config synthetic/branin/config.json --options options_files/options_example.txt !python /content/dragonfly/bin/dragonfly-script.py --config synthetic/branin/config.pb --options options_files/options_example.txt
Minimization : By default setting, dragonfly is set to maximize the function. But we can change that setting by usingthe max_or_min flag. You can check the example here.
!python dragonfly-script.py --config synthetic/branin/config.json --optionsoptions_files/options_example_for_minimisation.txt
Multi-fidelity Optimisation : Multi fidelity version of Branin benchmark is shown below
#multi-fidelity optmization !python dragonfly-script.py --config synthetic/branin/config_mf.json --options options_files/options_example.txt
You can find all types of optimization that can be run through cli here.
2. Dragonfly with Python Code
API’s for dragonfly are initialized in dragonfly/__init__.py and are defined in the dragonfly/apis folder. For example of using dragonfly API in python:
from dragonfly import minimise_function, maximise_function func = lambda x: x ** 4 - x**2 + 0.1 * x domain = [[-10, 10]] max_capital = 100 min_val, min_pt, history = minimise_function(func, domain, max_capital) print(min_val, min_pt) max_val, max_pt, history = maximise_function(lambda x: -func(x), domain, max_capital) print(max_val, max_pt)
Here,
func : It is the optimization function which is to be maximized.
domain : Over which the above func needs to be maximized.
max_capital : capital available for optimization(max. Number of available functions depends on cost or other constraints).
maximise_function : Returns the history with optimum value and the optimum point.
history.query_points : Represents the points evaluated by algorithm
history.query_vals : It contains the function values.
You can check other examples as well :
- Branin Example for maximization
- Hartmann Benchmark
- Example of discrete euclidean domains
- Example of Constrained Hartmann Benchmark
- borehole constrained example
- use case for random forest regression and gradient boosted regression
- A demo of neural architecture search using the NASBOT algorithm on an MLP (Multi-Layer Perceptron) architecture search problem
Similarly, you call use multiple objective optimization API. The basic structure of it is shown below :
from dragonfly import multiobjective_maximise_functions pareto_max_values, pareto_points, history = multiobjective_maximise_functions(funcs, domain, max_capital)
where,
funcs : list of functions to be maximised,
pareto_values : Pareto optimal function values
pareto_points : The corresponding points in domain.
You can check multi-objective branin example and multi-objective hartmann example.
You can check all other optimization available in dragonfly for python here.
3. Dragonfly with Ask-tell
It enables step-by-step optimization by calculating the next point to be evaluated in a loop of BO.
Two main components of this interface are:
- A function caller
- EuclideanFunctionCaller : Used when domain is limited to euclidean spaces
- CPFunctionCaller : else use this caller
- An optimizer. Here, <domain> is replaced by Euclidean or CP(depending on the domain).
- <domain>GPBandit
- <domain>GAOptimiser
- <domain>RandomOptimiser
An example of it is shown below:
from dragonfly.exd import domains
from dragonfly.exd.experiment_caller import CPFunctionCaller, EuclideanFunctionCaller
from dragonfly.opt import random_optimiser, cp_ga_optimiser, gp_bandit
max_capital = 100
objective = lambda x: x[0] ** 4 - x[0]**2 + 0.1 * x[0]
domain = domains.EuclideanDomain([[-10, 10]])
func_caller = EuclideanFunctionCaller(None, domain)
opt = gp_bandit.EuclideanGPBandit(func_caller, ask_tell_mode=True)
#call initialise() on the created optimizer to begin using the interface
opt.initialise()
for i in range(max_capital):
#ask returns the next point to be evaluated in a numpy array.
x = opt.ask()
y = objective(x)
print("x:", x, ", y:", y)
#Once the point is evaluated by calling the objective function,
#tell the point back to the optimiser
opt.tell([(x, y)])
The above example represents the maximization problem. For minimization, take the negative of the objective function and rest of the procedure is same.
Check all the details here.
Conclusion
In this post, we have given an overview of Dragonfly, an optimized framework for Bayesian Optimization that is scalable and robust as well. This framework is built keeping in mind the optimization of expensive optimization functions. It outperformed many existing systems available for BO.
Official codes, docs & Tutorial are available at :
