Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1355 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.3236. Min Act: -0.0000

Data Index: 1078090 (C4 (Web Text))

Max Activating Token Index: 348

Text #1

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.3103. Min Act: -0.0000

Data Index: 32668 (C4 (Web Text))

Max Activating Token Index: 220

Text #2

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.3343. Min Act: -0.0000

Data Index: 1175528 (C4 (Web Text))

Max Activating Token Index: 314

Text #3

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2902. Min Act: -0.0000

Data Index: 1085135 (C4 (Web Text))

Max Activating Token Index: 713

Text #4

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2693. Min Act: -0.0000

Data Index: 333801 (C4 (Web Text))

Max Activating Token Index: 786

Text #5

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.3038. Min Act: -0.0000

Data Index: 708800 (C4 (Web Text))

Max Activating Token Index: 577

Text #6

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2771. Min Act: -0.0000

Data Index: 1335904 (C4 (Web Text))

Max Activating Token Index: 208

Text #7

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2975. Min Act: -0.0000

Data Index: 43836 (C4 (Web Text))

Max Activating Token Index: 912

Text #8

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2722. Min Act: -0.0000

Data Index: 990575 (C4 (Web Text))

Max Activating Token Index: 927

Text #9

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2653. Min Act: -0.0000

Data Index: 276266 (C4 (Web Text))

Max Activating Token Index: 261

Text #10

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2904. Min Act: -0.0000

Data Index: 1133300 (C4 (Web Text))

Max Activating Token Index: 91

Text #11

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2717. Min Act: -0.0000

Data Index: 442498 (C4 (Web Text))

Max Activating Token Index: 524

Text #12

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2644. Min Act: -0.0000

Data Index: 1320195 (C4 (Web Text))

Max Activating Token Index: 146

Text #13

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2777. Min Act: -0.0000

Data Index: 29717 (C4 (Web Text))

Max Activating Token Index: 173

Text #14

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2786. Min Act: -0.0000

Data Index: 778894 (C4 (Web Text))

Max Activating Token Index: 923

Text #15

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2660. Min Act: -0.0000

Data Index: 915407 (C4 (Web Text))

Max Activating Token Index: 276

Text #16

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2548. Min Act: -0.0000

Data Index: 1062834 (C4 (Web Text))

Max Activating Token Index: 339

Text #17

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2771. Min Act: -0.0000

Data Index: 326137 (C4 (Web Text))

Max Activating Token Index: 89

Text #18

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2771. Min Act: -0.0000

Data Index: 249711 (C4 (Web Text))

Max Activating Token Index: 89

Text #19

Max Range: 0.3343. Min Range: -0.3343

Max Act: 0.2693. Min Act: -0.0000

Data Index: 540497 (C4 (Web Text))

Max Activating Token Index: 726