Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 476 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.5406. Min Act: -0.0000

Data Index: 733486 (C4 (Web Text))

Max Activating Token Index: 503

Text #1

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4650. Min Act: -0.0000

Data Index: 994042 (C4 (Web Text))

Max Activating Token Index: 66

Text #2

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4577. Min Act: -0.0000

Data Index: 887553 (C4 (Web Text))

Max Activating Token Index: 990

Text #3

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.5024. Min Act: -0.0000

Data Index: 379217 (C4 (Web Text))

Max Activating Token Index: 126

Text #4

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4618. Min Act: -0.0000

Data Index: 1011122 (C4 (Web Text))

Max Activating Token Index: 329

Text #5

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4646. Min Act: -0.0000

Data Index: 1140427 (C4 (Web Text))

Max Activating Token Index: 602

Text #6

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4489. Min Act: -0.0000

Data Index: 269051 (C4 (Web Text))

Max Activating Token Index: 434

Text #7

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4480. Min Act: -0.0000

Data Index: 877234 (C4 (Web Text))

Max Activating Token Index: 351

Text #8

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4304. Min Act: -0.0000

Data Index: 1032907 (C4 (Web Text))

Max Activating Token Index: 878

Text #9

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4110. Min Act: -0.0000

Data Index: 561953 (C4 (Web Text))

Max Activating Token Index: 664

Text #10

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4401. Min Act: -0.0000

Data Index: 664626 (C4 (Web Text))

Max Activating Token Index: 650

Text #11

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4282. Min Act: -0.0000

Data Index: 3177 (C4 (Web Text))

Max Activating Token Index: 746

Text #12

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4064. Min Act: -0.0000

Data Index: 949578 (C4 (Web Text))

Max Activating Token Index: 982

Text #13

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4137. Min Act: -0.0000

Data Index: 512567 (C4 (Web Text))

Max Activating Token Index: 300

Text #14

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4145. Min Act: -0.0000

Data Index: 715022 (C4 (Web Text))

Max Activating Token Index: 595

Text #15

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4309. Min Act: -0.0000

Data Index: 1239551 (C4 (Web Text))

Max Activating Token Index: 974

Text #16

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4381. Min Act: -0.0000

Data Index: 744797 (C4 (Web Text))

Max Activating Token Index: 950

Text #17

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4141. Min Act: -0.0000

Data Index: 371102 (C4 (Web Text))

Max Activating Token Index: 698

Text #18

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.3903. Min Act: -0.0000

Data Index: 219541 (C4 (Web Text))

Max Activating Token Index: 816

Text #19

Max Range: 0.5406. Min Range: -0.5406

Max Act: 0.4150. Min Act: -0.0000

Data Index: 1259393 (C4 (Web Text))

Max Activating Token Index: 741