Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 113 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2217. Min Range: -2.2217

Max Act: 2.2217. Min Act: -0.0000

Data Index: 1374385 (Python Code)

Max Activating Token Index: 331

Text #1

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.4830. Min Act: -0.0000

Data Index: 1454241 (Python Code)

Max Activating Token Index: 338

Text #2

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.6803. Min Act: -0.0000

Data Index: 1469890 (Python Code)

Max Activating Token Index: 1018

Text #3

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.5726. Min Act: -0.0000

Data Index: 1056326 (C4 (Web Text))

Max Activating Token Index: 176

Text #4

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.5155. Min Act: -0.0000

Data Index: 1505721 (Python Code)

Max Activating Token Index: 283

Text #5

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.4207. Min Act: -0.0000

Data Index: 1543376 (Python Code)

Max Activating Token Index: 862

Text #6

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.4084. Min Act: -0.0000

Data Index: 221572 (C4 (Web Text))

Max Activating Token Index: 611

Text #7

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.2024. Min Act: -0.0000

Data Index: 769371 (C4 (Web Text))

Max Activating Token Index: 439

Text #8

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.1959. Min Act: -0.0000

Data Index: 379915 (C4 (Web Text))

Max Activating Token Index: 85

Text #9

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.1460. Min Act: -0.0000

Data Index: 808528 (C4 (Web Text))

Max Activating Token Index: 871

Text #10

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.0820. Min Act: -0.0000

Data Index: 746212 (C4 (Web Text))

Max Activating Token Index: 88

Text #11

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.1183. Min Act: -0.0000

Data Index: 1624917 (Python Code)

Max Activating Token Index: 793

Text #12

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.1167. Min Act: -0.0000

Data Index: 328023 (C4 (Web Text))

Max Activating Token Index: 202

Text #13

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.1134. Min Act: -0.0000

Data Index: 476805 (C4 (Web Text))

Max Activating Token Index: 42

Text #14

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.0257. Min Act: -0.0000

Data Index: 593281 (C4 (Web Text))

Max Activating Token Index: 1018

Text #15

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.0892. Min Act: -0.0000

Data Index: 996179 (C4 (Web Text))

Max Activating Token Index: 442

Text #16

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.1688. Min Act: -0.0000

Data Index: 1311244 (C4 (Web Text))

Max Activating Token Index: 311

Text #17

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.0841. Min Act: -0.0000

Data Index: 1278507 (C4 (Web Text))

Max Activating Token Index: 363

Text #18

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.0080. Min Act: -0.0000

Data Index: 271845 (C4 (Web Text))

Max Activating Token Index: 441

Text #19

Max Range: 2.2217. Min Range: -2.2217

Max Act: 1.0801. Min Act: -0.0000

Data Index: 1057358 (C4 (Web Text))

Max Activating Token Index: 35