Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5869 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.9011. Min Act: -0.0000

Data Index: 1243077 (C4 (Web Text))

Max Activating Token Index: 874

Text #1

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.8880. Min Act: -0.0000

Data Index: 339184 (C4 (Web Text))

Max Activating Token Index: 790

Text #2

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.6000. Min Act: -0.0000

Data Index: 423234 (C4 (Web Text))

Max Activating Token Index: 90

Text #3

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.5946. Min Act: -0.0000

Data Index: 674499 (C4 (Web Text))

Max Activating Token Index: 425

Text #4

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.6008. Min Act: -0.0000

Data Index: 612133 (C4 (Web Text))

Max Activating Token Index: 790

Text #5

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.5177. Min Act: -0.0000

Data Index: 1315772 (C4 (Web Text))

Max Activating Token Index: 210

Text #6

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.5046. Min Act: -0.0000

Data Index: 305610 (C4 (Web Text))

Max Activating Token Index: 991

Text #7

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.5011. Min Act: -0.0000

Data Index: 744272 (C4 (Web Text))

Max Activating Token Index: 745

Text #8

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4985. Min Act: 0.0000

Data Index: 523404 (C4 (Web Text))

Max Activating Token Index: 275

Text #9

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4977. Min Act: -0.0000

Data Index: 1261755 (C4 (Web Text))

Max Activating Token Index: 462

Text #10

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4997. Min Act: -0.0000

Data Index: 833953 (C4 (Web Text))

Max Activating Token Index: 357

Text #11

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4066. Min Act: -0.0000

Data Index: 968332 (C4 (Web Text))

Max Activating Token Index: 245

Text #12

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4356. Min Act: -0.0000

Data Index: 628143 (C4 (Web Text))

Max Activating Token Index: 246

Text #13

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4279. Min Act: -0.0000

Data Index: 187822 (C4 (Web Text))

Max Activating Token Index: 424

Text #14

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4216. Min Act: -0.0000

Data Index: 1347761 (C4 (Web Text))

Max Activating Token Index: 394

Text #15

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4387. Min Act: -0.0000

Data Index: 8975 (C4 (Web Text))

Max Activating Token Index: 40

Text #16

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4356. Min Act: -0.0000

Data Index: 506039 (C4 (Web Text))

Max Activating Token Index: 767

Text #17

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4166. Min Act: 0.0000

Data Index: 395237 (C4 (Web Text))

Max Activating Token Index: 760

Text #18

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4162. Min Act: -0.0000

Data Index: 1066749 (C4 (Web Text))

Max Activating Token Index: 990

Text #19

Max Range: 0.9011. Min Range: -0.9011

Max Act: 0.4221. Min Act: -0.0000

Data Index: 1049197 (C4 (Web Text))

Max Activating Token Index: 838