Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 64 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.3494. Min Act: -0.0000

Data Index: 1512362 (Python Code)

Max Activating Token Index: 119

Text #1

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.3392. Min Act: -0.0000

Data Index: 1603930 (Python Code)

Max Activating Token Index: 210

Text #2

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.3314. Min Act: -0.0000

Data Index: 1633537 (Python Code)

Max Activating Token Index: 41

Text #3

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.3334. Min Act: -0.0000

Data Index: 1405177 (Python Code)

Max Activating Token Index: 84

Text #4

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.3169. Min Act: -0.0000

Data Index: 1583204 (Python Code)

Max Activating Token Index: 630

Text #5

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2836. Min Act: -0.0000

Data Index: 1527177 (Python Code)

Max Activating Token Index: 471

Text #6

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.3104. Min Act: -0.0000

Data Index: 110443 (C4 (Web Text))

Max Activating Token Index: 835

Text #7

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2970. Min Act: -0.0000

Data Index: 1372353 (Python Code)

Max Activating Token Index: 932

Text #8

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2857. Min Act: -0.0000

Data Index: 1448375 (Python Code)

Max Activating Token Index: 555

Text #9

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2807. Min Act: -0.0000

Data Index: 529076 (C4 (Web Text))

Max Activating Token Index: 61

Text #10

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2905. Min Act: -0.0000

Data Index: 456980 (C4 (Web Text))

Max Activating Token Index: 191

Text #11

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2888. Min Act: -0.0000

Data Index: 1646269 (Python Code)

Max Activating Token Index: 146

Text #12

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2719. Min Act: -0.0000

Data Index: 1627199 (Python Code)

Max Activating Token Index: 818

Text #13

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2678. Min Act: -0.0000

Data Index: 1527208 (Python Code)

Max Activating Token Index: 752

Text #14

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2592. Min Act: -0.0000

Data Index: 1608087 (Python Code)

Max Activating Token Index: 883

Text #15

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2800. Min Act: -0.0000

Data Index: 1595941 (Python Code)

Max Activating Token Index: 150

Text #16

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2442. Min Act: -0.0000

Data Index: 1368786 (Python Code)

Max Activating Token Index: 638

Text #17

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2633. Min Act: -0.0000

Data Index: 1195893 (C4 (Web Text))

Max Activating Token Index: 876

Text #18

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2558. Min Act: -0.0000

Data Index: 1486027 (Python Code)

Max Activating Token Index: 64

Text #19

Max Range: 0.3494. Min Range: -0.3494

Max Act: 0.2511. Min Act: -0.0000

Data Index: 1635319 (Python Code)

Max Activating Token Index: 337