Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4605 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2138. Min Range: -1.2138

Max Act: 1.1831. Min Act: -0.0000

Data Index: 74836 (C4 (Web Text))

Max Activating Token Index: 445

Text #1

Max Range: 1.2138. Min Range: -1.2138

Max Act: 1.2138. Min Act: -0.0000

Data Index: 761753 (C4 (Web Text))

Max Activating Token Index: 747

Text #2

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.8365. Min Act: -0.0000

Data Index: 718280 (C4 (Web Text))

Max Activating Token Index: 343

Text #3

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.8659. Min Act: -0.0000

Data Index: 1063555 (C4 (Web Text))

Max Activating Token Index: 524

Text #4

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.8045. Min Act: -0.0000

Data Index: 728148 (C4 (Web Text))

Max Activating Token Index: 871

Text #5

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.8201. Min Act: -0.0000

Data Index: 240376 (C4 (Web Text))

Max Activating Token Index: 326

Text #6

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.6084. Min Act: -0.0000

Data Index: 103224 (C4 (Web Text))

Max Activating Token Index: 177

Text #7

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.6082. Min Act: -0.0000

Data Index: 1320197 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.6156. Min Act: -0.0000

Data Index: 1210492 (C4 (Web Text))

Max Activating Token Index: 926

Text #9

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5672. Min Act: -0.0000

Data Index: 694685 (C4 (Web Text))

Max Activating Token Index: 470

Text #10

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5688. Min Act: -0.0000

Data Index: 1283277 (C4 (Web Text))

Max Activating Token Index: 570

Text #11

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5811. Min Act: -0.0000

Data Index: 725517 (C4 (Web Text))

Max Activating Token Index: 607

Text #12

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5806. Min Act: -0.0000

Data Index: 9778 (C4 (Web Text))

Max Activating Token Index: 32

Text #13

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.6040. Min Act: -0.0000

Data Index: 858230 (C4 (Web Text))

Max Activating Token Index: 824

Text #14

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5551. Min Act: -0.0000

Data Index: 586082 (C4 (Web Text))

Max Activating Token Index: 255

Text #15

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5530. Min Act: -0.0000

Data Index: 770387 (C4 (Web Text))

Max Activating Token Index: 416

Text #16

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.4961. Min Act: -0.0000

Data Index: 1308271 (C4 (Web Text))

Max Activating Token Index: 971

Text #17

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5087. Min Act: -0.0000

Data Index: 921168 (C4 (Web Text))

Max Activating Token Index: 219

Text #18

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5134. Min Act: -0.0000

Data Index: 213789 (C4 (Web Text))

Max Activating Token Index: 427

Text #19

Max Range: 1.2138. Min Range: -1.2138

Max Act: 0.5211. Min Act: -0.0000

Data Index: 703474 (C4 (Web Text))

Max Activating Token Index: 894