Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2396 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.7507. Min Range: -2.7507

Max Act: 2.7507. Min Act: -0.0000

Data Index: 583177 (C4 (Web Text))

Max Activating Token Index: 35

Text #1

Max Range: 2.7507. Min Range: -2.7507

Max Act: 2.0719. Min Act: -0.0000

Data Index: 1336085 (C4 (Web Text))

Max Activating Token Index: 435

Text #2

Max Range: 2.7507. Min Range: -2.7507

Max Act: 2.0627. Min Act: -0.0000

Data Index: 1289407 (C4 (Web Text))

Max Activating Token Index: 961

Text #3

Max Range: 2.7507. Min Range: -2.7507

Max Act: 2.0114. Min Act: -0.0000

Data Index: 410221 (C4 (Web Text))

Max Activating Token Index: 497

Text #4

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.9590. Min Act: -0.0000

Data Index: 435389 (C4 (Web Text))

Max Activating Token Index: 947

Text #5

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.7208. Min Act: -0.0000

Data Index: 568702 (C4 (Web Text))

Max Activating Token Index: 627

Text #6

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.5534. Min Act: -0.0000

Data Index: 1183889 (C4 (Web Text))

Max Activating Token Index: 875

Text #7

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.6560. Min Act: -0.0000

Data Index: 730884 (C4 (Web Text))

Max Activating Token Index: 378

Text #8

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.6943. Min Act: -0.0000

Data Index: 214918 (C4 (Web Text))

Max Activating Token Index: 826

Text #9

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.5494. Min Act: -0.0000

Data Index: 545468 (C4 (Web Text))

Max Activating Token Index: 50

Text #10

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.5560. Min Act: -0.0000

Data Index: 859623 (C4 (Web Text))

Max Activating Token Index: 608

Text #11

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.5757. Min Act: -0.0000

Data Index: 1167590 (C4 (Web Text))

Max Activating Token Index: 485

Text #12

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.4791. Min Act: -0.0000

Data Index: 1040877 (C4 (Web Text))

Max Activating Token Index: 387

Text #13

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.3467. Min Act: -0.0000

Data Index: 597629 (C4 (Web Text))

Max Activating Token Index: 483

Text #14

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.4225. Min Act: -0.0000

Data Index: 756421 (C4 (Web Text))

Max Activating Token Index: 576

Text #15

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.4031. Min Act: -0.0000

Data Index: 1131390 (C4 (Web Text))

Max Activating Token Index: 762

Text #16

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.3827. Min Act: -0.0000

Data Index: 927524 (C4 (Web Text))

Max Activating Token Index: 513

Text #17

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.4718. Min Act: -0.0000

Data Index: 1261156 (C4 (Web Text))

Max Activating Token Index: 974

Text #18

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.2794. Min Act: -0.0000

Data Index: 384871 (C4 (Web Text))

Max Activating Token Index: 737

Text #19

Max Range: 2.7507. Min Range: -2.7507

Max Act: 1.3177. Min Act: -0.0000

Data Index: 1045957 (C4 (Web Text))

Max Activating Token Index: 557