Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.4176. Min Act: -0.0000

Data Index: 918650 (C4 (Web Text))

Max Activating Token Index: 618

Text #1

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3766. Min Act: -0.0000

Data Index: 1173148 (C4 (Web Text))

Max Activating Token Index: 825

Text #2

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3342. Min Act: -0.0000

Data Index: 61886 (C4 (Web Text))

Max Activating Token Index: 339

Text #3

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2823. Min Act: -0.0000

Data Index: 358141 (C4 (Web Text))

Max Activating Token Index: 135

Text #4

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3067. Min Act: -0.0000

Data Index: 1110595 (C4 (Web Text))

Max Activating Token Index: 686

Text #5

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3153. Min Act: -0.0000

Data Index: 210557 (C4 (Web Text))

Max Activating Token Index: 234

Text #6

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2839. Min Act: -0.0000

Data Index: 1359570 (C4 (Web Text))

Max Activating Token Index: 654

Text #7

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3006. Min Act: -0.0000

Data Index: 64629 (C4 (Web Text))

Max Activating Token Index: 37

Text #8

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3174. Min Act: -0.0000

Data Index: 1238100 (C4 (Web Text))

Max Activating Token Index: 912

Text #9

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.3054. Min Act: -0.0000

Data Index: 1019038 (C4 (Web Text))

Max Activating Token Index: 91

Text #10

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2934. Min Act: -0.0000

Data Index: 541099 (C4 (Web Text))

Max Activating Token Index: 966

Text #11

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2798. Min Act: -0.0000

Data Index: 883609 (C4 (Web Text))

Max Activating Token Index: 343

Text #12

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2629. Min Act: -0.0000

Data Index: 347051 (C4 (Web Text))

Max Activating Token Index: 539

Text #13

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2810. Min Act: -0.0000

Data Index: 796598 (C4 (Web Text))

Max Activating Token Index: 329

Text #14

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2676. Min Act: -0.0000

Data Index: 363985 (C4 (Web Text))

Max Activating Token Index: 113

Text #15

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2841. Min Act: -0.0000

Data Index: 645660 (C4 (Web Text))

Max Activating Token Index: 752

Text #16

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2663. Min Act: -0.0000

Data Index: 1158523 (C4 (Web Text))

Max Activating Token Index: 622

Text #17

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2681. Min Act: -0.0000

Data Index: 743143 (C4 (Web Text))

Max Activating Token Index: 186

Text #18

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2562. Min Act: -0.0000

Data Index: 423366 (C4 (Web Text))

Max Activating Token Index: 366

Text #19

Max Range: 0.4176. Min Range: -0.4176

Max Act: 0.2617. Min Act: -0.0000

Data Index: 627051 (C4 (Web Text))

Max Activating Token Index: 103