Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5011 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.2870. Min Range: -3.2870

Max Act: 3.2870. Min Act: -0.0000

Data Index: 438236 (C4 (Web Text))

Max Activating Token Index: 101

Text #1

Max Range: 3.2870. Min Range: -3.2870

Max Act: 3.0601. Min Act: -0.0000

Data Index: 1441223 (Python Code)

Max Activating Token Index: 222

Text #2

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.8530. Min Act: -0.0000

Data Index: 933852 (C4 (Web Text))

Max Activating Token Index: 711

Text #3

Max Range: 3.2870. Min Range: -3.2870

Max Act: 3.0321. Min Act: -0.0000

Data Index: 1210612 (C4 (Web Text))

Max Activating Token Index: 150

Text #4

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.8009. Min Act: -0.0000

Data Index: 165112 (C4 (Web Text))

Max Activating Token Index: 579

Text #5

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5775. Min Act: -0.0000

Data Index: 925176 (C4 (Web Text))

Max Activating Token Index: 119

Text #6

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.6534. Min Act: -0.0000

Data Index: 1348994 (C4 (Web Text))

Max Activating Token Index: 205

Text #7

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5227. Min Act: -0.0000

Data Index: 759835 (C4 (Web Text))

Max Activating Token Index: 84

Text #8

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5781. Min Act: -0.0000

Data Index: 363148 (C4 (Web Text))

Max Activating Token Index: 236

Text #9

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5497. Min Act: -0.0000

Data Index: 130912 (C4 (Web Text))

Max Activating Token Index: 756

Text #10

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5137. Min Act: -0.0000

Data Index: 1185264 (C4 (Web Text))

Max Activating Token Index: 1018

Text #11

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.3441. Min Act: -0.0000

Data Index: 1045654 (C4 (Web Text))

Max Activating Token Index: 356

Text #12

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5906. Min Act: -0.0000

Data Index: 1123697 (C4 (Web Text))

Max Activating Token Index: 966

Text #13

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.5676. Min Act: -0.0000

Data Index: 495403 (C4 (Web Text))

Max Activating Token Index: 790

Text #14

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.4166. Min Act: -0.0000

Data Index: 950681 (C4 (Web Text))

Max Activating Token Index: 269

Text #15

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.4855. Min Act: -0.0000

Data Index: 165869 (C4 (Web Text))

Max Activating Token Index: 259

Text #16

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.2090. Min Act: -0.0000

Data Index: 967799 (C4 (Web Text))

Max Activating Token Index: 351

Text #17

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.2981. Min Act: -0.0000

Data Index: 311750 (C4 (Web Text))

Max Activating Token Index: 542

Text #18

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.2433. Min Act: -0.0000

Data Index: 337576 (C4 (Web Text))

Max Activating Token Index: 7

Text #19

Max Range: 3.2870. Min Range: -3.2870

Max Act: 2.1371. Min Act: -0.0000

Data Index: 832150 (C4 (Web Text))

Max Activating Token Index: 590