Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2434 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2167. Min Range: -1.2167

Max Act: 1.1928. Min Act: -0.0000

Data Index: 853253 (C4 (Web Text))

Max Activating Token Index: 969

Text #1

Max Range: 1.2167. Min Range: -1.2167

Max Act: 1.2167. Min Act: -0.0000

Data Index: 176242 (C4 (Web Text))

Max Activating Token Index: 883

Text #2

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.9174. Min Act: -0.0000

Data Index: 328572 (C4 (Web Text))

Max Activating Token Index: 821

Text #3

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.8884. Min Act: -0.0000

Data Index: 338582 (C4 (Web Text))

Max Activating Token Index: 604

Text #4

Max Range: 1.2167. Min Range: -1.2167

Max Act: 1.0802. Min Act: -0.0000

Data Index: 201791 (C4 (Web Text))

Max Activating Token Index: 758

Text #5

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.8889. Min Act: -0.0000

Data Index: 30961 (C4 (Web Text))

Max Activating Token Index: 578

Text #6

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.8385. Min Act: -0.0000

Data Index: 339486 (C4 (Web Text))

Max Activating Token Index: 726

Text #7

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7750. Min Act: -0.0000

Data Index: 1075234 (C4 (Web Text))

Max Activating Token Index: 584

Text #8

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7199. Min Act: -0.0000

Data Index: 795599 (C4 (Web Text))

Max Activating Token Index: 578

Text #9

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.8095. Min Act: -0.0000

Data Index: 131605 (C4 (Web Text))

Max Activating Token Index: 747

Text #10

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7467. Min Act: -0.0000

Data Index: 297795 (C4 (Web Text))

Max Activating Token Index: 624

Text #11

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7517. Min Act: -0.0000

Data Index: 1221680 (C4 (Web Text))

Max Activating Token Index: 460

Text #12

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7470. Min Act: -0.0000

Data Index: 1050912 (C4 (Web Text))

Max Activating Token Index: 127

Text #13

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7388. Min Act: -0.0000

Data Index: 907970 (C4 (Web Text))

Max Activating Token Index: 743

Text #14

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.6808. Min Act: -0.0000

Data Index: 141303 (C4 (Web Text))

Max Activating Token Index: 612

Text #15

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.7093. Min Act: -0.0000

Data Index: 470859 (C4 (Web Text))

Max Activating Token Index: 119

Text #16

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.6817. Min Act: -0.0000

Data Index: 1000419 (C4 (Web Text))

Max Activating Token Index: 140

Text #17

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.6241. Min Act: -0.0000

Data Index: 510460 (C4 (Web Text))

Max Activating Token Index: 990

Text #18

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.6223. Min Act: -0.0000

Data Index: 812558 (C4 (Web Text))

Max Activating Token Index: 872

Text #19

Max Range: 1.2167. Min Range: -1.2167

Max Act: 0.6299. Min Act: -0.0000

Data Index: 795555 (C4 (Web Text))

Max Activating Token Index: 1022